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ABSTRACT 



In today's complex CBT/CAI market, flashy presentations 
frequently prove the most important purchasing element while 
instructional design and content take a back seat to form. 
Courseware purchasers must base decisions upon either a 
vendor's presentation or some published evaluator rating. 
Unfortunately, these are almost certain to be biased by 
irrelevant factors such as color graphics or presentation 
speed. The CITAR courseware evalaution model (CCCEM) 
emphasizes the instructional components of such courseware 
rather than the "bells & whistles" of the associated 
technology and provides descriptive information on more than 
300 courseware components. Additionally, over 200 item-level 
tallies are synthesized into scores that may be used to 
compare similar packages on their Instruction, Management, 
Physical and Presentation aspects. A study of the model's 
consistency found on average that 7 of 8 evaluators tend to 
agree on items and scores, and that ICC and test-retest 
reliabilities for key scores were near .70. A comparison of 
traditional perceptual evaluations for the same courseware 
produced reliabilities averaging over .40 less than the CCCEM 
model, and agreements averaging about .20 less. The results 
of this study suggest that the CCCEM fills the need of 
courseware purchasers for an objective, generic measure. 
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EXECUTIVE SUMMARY 

Funded by a grant from the Westinghouse Corporation, the 
Center for Interactive Technologies, Applications and Re- 
search (CITAR) at the University of South Florida initiated 
in July, 1986, an attempt to develop an objective, generic 
model for the evalaution of computer based courseware. The 
CITAR Computer Courseware Evaluation Model (CCCEM) attempts 
to break conceptually complex concepts such as instruction, 
management and user interface into component pieces small 
enough to be objectively defined. In this way, both the ex- 
istence and prevalence of various content and technological 
aspects of computer courseware may be compared across prod- 
ucts. 

The CCCEM contains information on over 250 facets of CBT 
package courseware. Faced with this plethora of information 
the harried consumer will most likely base his/er purchasing 
decision almost entirely on an unreliable evaluator rating. 
Unfortunately, synthesizing so many diverse phenomena is 
quite difficult, therefore overall ratings based soley on 
evaluator interpretations more likely indicate personal bi- 
ases than TRUE differences in software effectiveness. CCCEM 
offers an alternative in the form of absolute scores on 
which the relative efficiency of different educational soft- 
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ware may be compared. The items included in these scores 
may be conceived of as stylistic or generic in the sense 
that their presence should associate with improved learning 
for almost any subject area. Further, the presence of more 
generic items should associate with "greater" learning than 
the presence of fewer such items. 

Average agreement among raters across the several major 
areas of content (Physical, Presentation, Management and In- 
struction) produced interrater agreement percentages showing 
that approximately nine of ten raters code the same (above 
80% agreement). Thus, overall agreement appears quite ade- 
quate. Certain specific items (i.e. in Questioning, Se- 
quencing and Response Judging) produced lower percentages, 
showing agreement among as few as three of four raters. 

Estimates of reliability (an instrument's ability to con- 
sistently discriminate among CBT packages) were adequate for 
Efficacy, Management and Instruction scores (circa .70), and 
acceptable for Physical scores (.55). However, the Presen- 
tation score is not adequately reliable for decision purpos- 
es (.15). The modest reliabilities exhibited by the Presen- 
tation and Physical scores result largely from the limited 
variability found among CBT packages in these areas. AS 
software becomes more sophisticated and efficacious, one 
would expect variability and therefore reliability to in- 
crease. Given the current levels of variability among CBT 



packages, it appears appropriate to separate the Outstanding 
from the Excellent, the Moderate, and the Questionable, on 
Efficacy, Instruction, Management and Physical characteris- 
tics. 

Additional study, directly comparing simple rater percep- 
tions with the relatively objective CCCEM scores suggests 
that the CCCEM exhibits a substantial advantage both in 
agreement and reliability, in fact, CCCEM reliabilities av- 
eraged .47 higher than rater perceptions for the same ques- 
tions. 
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BACKGROUND AND METHODS 

This document presents the results of a study conducted to 
determine the operational consistency (reliability) of the 
Center for Interactive Technologies, Applications and Re- 
search (CITAR) Computer Courseware Evaluation Model (CCCEM). 
By definition, a reliable instrument exhibits stability, 
consistency, dependability and small errors of measurement 
on the characteristic begin measured. Although all measures 
include some measurement error, judgements made by humans 
are especially plagued by this problem. Thus, clearly de- 
lineated judgement criteria become a primary concern of in- 
strumentation involving human decisions. This study exam- 
ines the consistency with which judgements on 10 pieces of 
educational software are made by a set of four raters using 
CCCEM. 

Reliability may be viewed as the consistency with which 
an instrument differentiates among a group of targets (CBT 
packages). Put simply, the consistency with which the same 
relative rankings are assigned to a specific group of tar- 
gets by different raters. A major source of error for ob- 
servation instruments is disagreement among raters, thus 
this issue was considered separately from the overall reli- 
ability question to isolate its influence. Interrater 
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agreement may be defined as the extent to which two or more 
observers, working independently, agree on which phenomena 
occur to what degree in the target of interest. 

Mitchell (1979) claims the most common index of rater 
consistency in observational studies to be the interobserver 
agreement percentage. This technique, although an excellent 
measure of *.he absolute magnitude of one type of error (ob- 
server disagreement), provides no information about an in- 
strument's ability to differentiate among targets, there- 
fore, extremely high interrater agreement percentages may 
associate with very low reliabilities and vice versa. Al- 
though oft criticized as a measure of consistency (see Tows- 
topiat, 1984; Frick & Semmel, 1978), mean item level per- 
centage of agreement provides an accurate and readily 
interpretable measure of rater consistency for the data pro- 
duced by the CCCEM model, particularly at the item level. 
Therefore, mean percentages of agreement were used as an es- 
timate of rater consistency. 

Although numerous techniques have been suggested for the 
investigation of observation instrument reliability, consen- 
sus appears to support the use of generalizability theory 
and the intraclass correlation coefficient (ICC) based on 
Analysis of variance (Mitchell, 1979; Shrout and Fleiss, 
1978; Tinsley and Weiss, 1975). Tinsley and Weiss (1975) 
support the use of the intraclass correlation (ICC) as an 
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estimate of rating scales reliability because it permits the 
inclusion or exclusion of the between-rater variance as part 
of the error variance (WMS) and because it allows an estima- 
tion of the precision of the reliability coefficient (p. 
363). 

Since every statistical estimate contains unique biases 
(Cook and Campbell, 1984) , a second, equally conservative 
estimate of reliability was also computed: average test/re- 
test among raters. By taking the average correlation of 
scores obtained among CBT packages for each rater with those 
of every other rater, this estimate indicates the degree to 
which different raters rank the same CBT packages in the 
same order, and as with the ICC f includes almost every 
source of variance in the error term. 

To estimate the reliability of CCCEM, one must asses its 
ability to consistently discriminate amoag various CBT pack- 
ages differing on items of interest. It should also fail to 
differentiate among those CBT packages not differing of 
items of interest. The issue therefore becomes: Which items 
are of interest? 

Assuming at least two major market segments for CBT pack- 
ages (managers/teachers and students), different aspects of 
these subscores interest different audiences. For example, 
management systems appeal more to teachers/managers than to 
students, while presentation and instructional aspects ap- 
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peal to both audiences. In addition, at least two major 
marketing considerations are (1) Instructional effectiveness 
and (2) Cost/Appeal/Physical characteristics. For example, 
although not necessarily more instructionally effective for 
highly motivated students, "Bells and Whistles" such as col- 
or graphics and animation may stimulate the interest of ret- 
icent learners and perhaps even appeal to the jaded tastes 
of experienced teachers/managers. 

Five major scores were created from CCCEM items to ser- 
vice the needs of varying audiences: 

1. Descriptive (not scalable) 

a) All specific characteristics of CBT package's, 

2. Instruction 

a) Characteristics of the learning environment, for 
instance, questioning, problem solving or gaming, 

3. Management 

a) Characteristics of the management system such as 
group level analysis and student performance re- 
ports . 

4 . Presentation 

a) Characteristics of physical interface between stu- 
dent and CBT package. 

5. Physical 

a) Characteristics of hardware and software. 



Each subscore relates the particular CBT pacKage against a 
hypothetical optimum CBT package regarding both software so- 
phistication and instructional effectiveness, 

ISSUES ARISING IN SCALE DEVELOPMENT AND INTERRATER AGREEMENT 
Instrument validation requires both that items vary across 
CBT packages, and that this variability be related to great- 
er and lesser instructional ef f ectiveness . Two CTBs may 
differ in one area {e.g. hardware or management systems) but 
not in others (e.g instruction techniques). In addition, 
various "differences" within each cell of the CCCEM model 
will impact instructional effectiveness and target audience 
appeal in varying ways. Failure to weigh such considera- 
tions could substantially reduce both the usefulness and 
marketability of CCCEM 1 s evaluations. 

Scalable items within the CCCEM model are descrioed in 
one of three ways. First is a binary Yes/No, defining exis- 
tence or non-existence. Secondly, a scale including four 
points is used to relate the necessity of a particular char- 
acteristic to the use of a package. This scale is designat- 
ed RONSI with points representing respectively: (R)equired, 
(O)ptional, {'X)ot (S)upported or (I)ndeterminate. Most 
RONSI d^ta are collected directly from vendor supplied ma- 
terials. Thirdly, a four point Likert-type scale d scribes 
* -ntity of or extent to which a characteristic appears 
CBT package. The items are designated ESON and rep- 
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resent respectively: (E)xtensively , (S) ignif icantly , 
(O)ccasionally or (N)egligible. 

The validity of an instrument depends upon relatively op- 
timal weightings of item and subscores in the creation of 
composite scores. For instance, if one item (i.e. defined 
objectives) is scored on a ycs/no basis and assigned values 
of 0 for no and 1 for yes f while another (i.e. use of sever- 
al fonts) is scored using ESON and assigned values of 0 for 
Negligible and 3 for Extensively, it is obvious that more 
than one font would influence score variability to a greater 
degree than the presence or absence of defined objectives. 
This, although the latter is far more important both to in- 
struction design and implementation. Thus, the weights as- 
signed to each item and subscore were estimated by a panel 
of "experts". These weightings may later be validated by 
studies of CBT package effectiveness. 

Secondly, the assignment of values to responses for RONSI 
and ESON may differ from item to item. 



ESON 

Page Turning 112 0 

Tutorials 2 2 10 
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The preceding assigns different values to different frequen- 
cies of occurrence for two ESON scaled items. Thus, the Ex-> 
tensive or Significant use of tutorials is given twice the 
value of its Occasional occurrence/ while the Extensive or 
Significant use of page turning receives only half the value 
of its Occasional occurrence. Although these issues contain 
statistical aspects, they must largely be solved by expert 



Maximum scores represent what to the best current knowledge 
are relatively OPTIMAL CBT package learning environments. 
Four of the five previously mentioned subscores are scala- 



1. Instruction - consisting of 115 scalable items, 6 ma- 
jor subscales and 16 minor subscales, 

2. Management - consisting of 51 scalable items and 9 
subscales , 

3. Presentation - consisting of 31 scalable items and 6 
subscales and 

4. Physical - consisting of 19 scalable items and 3 
subscales . 

These scales are amenable to combination into a total score 
which may be called: Efficacy. 

During May and June, 1987, five meetings were held with 
"experts" representing the following interest groups: man- 



opinion. 
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agement, instruction design, software evaluators and meas- 
urement. Each meeting included at least five such experts. 
During this time, item scalings were tentatively determined 
for each of the CCCEM items. In addition, subscore composi 
tion, levels of weightings (item, minor subscale, major 
scale) were tentatively defined. Tables 1 and 2 list subs- 
cales, weighting and scoring characteristics of the four ma 
jor submeasures, respectively: Instruction, Management, 
Presentation and Physical. 
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TABLE 1 

Components of Instruction Scores 



Major and Minor # Max Mean item 

Subscales Items Wt/Scr Weight 

PLANNING* 



1. Student Document 


5 


4 


0.80 


2. Task Analysis 


1 


6 


6.00 




6 


10 


1.67 


OBJECTIVES 








1. Defined Goals 


2 


30 


15.00 


2. Defined Outcomes 


6 


40 


6.67 


3. Defined Processes 


6 


5 


0.83 


sub totals 


20 


75 


3.75 


TECHNIQUES 








1 . Instruction 








Techniques 


10 


60 


6.00 


2. Physical Present 


4 


35 


8.75 


3. Timing 


5 


5 


1.00 


sub totals 


19 


100 


5.26 


INTERACTION 








1. Judging incorrect 








responses 


10 


35 


3 . 50 


2. Testing 


3 


25 


8.25 


3. Interaction 


16 


5 


0.31 


4. Questioning 


11 


25 


2.27 


5. Sequencing 


14 


20 


1.43 


sub totals 


54 


110 


2.04 


RESOURCE SCOPE 








1. Intrinsic 


5 


50 


10.00 


2. Extrinsic 


8 


20 


2.50 


3. Supplemental 


8 


10 


1.25 


sub totals 


21 


80 


3.81 


CONTENT ASSESSMENT 


1 


25 


25.00 


Totals 


115 


385 


3.34 



* Items comprising each scale and subscale are 
detailed in Appendix B. 





TABLE 2 








Components of Management, Presentation and 


Physical Scores 




Minor 


# 


Max 


Mean Item 




Subscales 


Items 


Wt/Scr 


Weights 




MANAGEMENT 










1 . Documentation 


8 


10 


1.25 




2. Record Keeping 


6 


25 


3.57 




3, Data Analysis/ 










Entry 


7 


20 


2.86 




4. Reports 


3 


15 


5.00 




5. Media 


4 


15 


3.75 




6. Access 


5 


10 


2.00 




7. Security 


2 


10 


5.00 




8. Learning 










Prescriptions 


5 


10 


2.00 




9. Instructor Browse/ 










Sequencing 


11 


10 


0.91 




Totals 


51 


125 


2.45 




PRESENTATION 










1. Interface 


6 


1 


0.15 




2. Text 


7 


10 


1.43 




3, Graphics 


7 


10 


1.43 




4. Sounds 


4 


10 


2.50 




5, Design Features 


4 


10 


2.50 




6. Text editing 


3 


1 


0.33 




Totals 


31 


42 


1.35 




PHYSICAL 










1. Screens 


7 


30 


4.29 




2. Peripherals 


11 


10 


0.91 




,3 • Concurrent 










applications 


1 


10 


10.00 




Totals 


19 


50 


2.63 




Note: Items comprising each 


scale and subscale 




are detailed in 


Appendix B. 
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SCALING ITEMS 

Within the subscales shown in Tables 1 and 2, each of the 
222 scalable CCCEM items was evaluated separately to deter- 
mine both its importance against all other items to which it 
relates, and an optimum application for each set of items 
(subscale). Every item receives a score from its rater with 
the following ranges: 

Yes No RONSI ESON 

1 0 1234 1234 

For most items, the lack of that item contributes nothing to 
either the management or instruction functions of the CBT 
package (No in the Yes/No dichotomy, Not Supported (NS) or 
Indeterminate under RONSI and Negligible for ESON* Thus, 
these were generally assigned a value of zero (0). For most 
items, Yes in the Yes/No dichotomy, Optional under RONSI, 
and either Occasional or Significant in ESON represents the 
optimal learning environment. For management subscales, Ex- 
tensive (under Eson) was frequently considered optimal. 
These scores generally received high values. Required 
(RONS~) or Extensive (ESON) use of any technique may be con- 
sidered sub-optimal in the learning environment, therefore, 
these usually received an intermediate value. 
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WEIGHTING 



Since the CCCEM contains numerous individual items , and 
since the number of items in a possible area of Instruction, 
Management or Physical/Presentation may or may not relate to 
the importance of that area in an overall perspective, each 
of the noted areas were assigned relative weights in the 
production of a total Efficacy score: Instruction (64%), 
Management (21%), Presentation (7%) and Physical (8%). 
Within each of the major scores produced by CCCEM, several 
subscales exist, containing different numbers of items. The 
Instruction score contains six major subscales (Planning, 
Objectives, Techniques, Interaction, Resource Scope and Con- 
tent Assessment) and 17 minor subscales. Each of the major 
subscales was weighted relative to the other five as fol- 
lows: Planning (2.5%), Objectives (19.5%), Techniques 
(26%), Interaction (28.6%), Resource Scope (20.8%) and Con- 
tent Assessment (6.5%). Planning and Content Assessment 
were assigned low weights due to their current incomplete 
status . 

Within each of the major subscales, its minor subscales 
was assigned a weight relative to each other minor subscale 
as shown in Tables 1 and 2. 

Within each of the minor subscales, each of the items 
comprising it was assigned a weight relative to each other 
item (see Appendix B for detailed item definition). For ex- 
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ample, the six items in the minor subscale Defined Outcomes 
were assigned the following values: Cognitive (2), Measured 
(2), Affective (1), Measured (2), Psychomotor (2), Measured 
(2). 

In creating scores and subscores, the relative weightings 
assigned to each item were multipled times the relative val- 
ues assigned to that item* The sum of values created by 
this process for each minor subscale was then computed and 
divided by the maximum possible score for the minor subscale 
(defined as an optimal situation - this need not include all 
items) and multiplied times the weighting assigned to that 
minor subscale. The sum of minor subscale values created in 
this fashion was used to compute major subscale values, and 
the sum of major subscale values taken as the value for that 
score. 

For example, the 10 items of the Instruction Techniques 
minor subscale of the Techniques major subscale of the In- 
struction score is composed of ESON items which are scaled 
and weighted as follows: 
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Item 



Weighting 



E S 0 N 



Page Turning 0 12 0 

Tutorial 12 2 0 

Drill & Practice 12 2 0 

Questioning 12 2 0 
Concurrent Training 2 2 10 

Problem Solving 12 2 0 

Simulation 2 2 10 

Modeling 12 2 0 

Gaming 112 0 

Inquiry 2 2 10 



0 
3 
2 
1 
4 
3 
4 
2 
2 
3 



The assumed optimum situation for Learning for this subscale 
would include the Occasional or Significant use of perhaps 
three or more higher weighted subscales, in addition to ex- 
pected items such as page turning or questioning. An "opti- 
mal" environment would score approximately 12 points. Thus, 
the total score obtained by each CBT package would be divid- 
ed oy 12 to determine the proportion of an optimal environ- 
ment contained within the CBT package. For example, a CBT 
package using Significant Page Turning (.5X0= 0.0) plus 
Occasional Drill & Practice (1.0 X 2 = 2) plus Significant 
Problem Solving (1.0 X 3 = 3) plus Occasional Simulation 
(0.5 X 4 = 2) would sum to 0.0 + 2.0 + 3.0 + 2.0 = 7.0 and 
would receive a score of 7.0 /12 = .583, or 58% of the opti- 
mal environment. This percentage would then be multiplied 
times the maximum subscale value for Instruction Technqiues 
(.58 X 60 = 35) and this value (35) would be summed with the 
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CBT package's score for Physical Presentation and Timing to 
create an Instrucational Techniques major subscale score 
(Table 1) . This score would then be combined with the other 
major subscales undur Instruction to derive a total Instruc- 
tion score. In addition, the proportion of "optimality" for 
a specific subscale, in this case .58, allows for absolute 
comparisons on a scale from zero to 100, where zero repre- 
sents "Useless" and 100, "Optimum". 

Interrater Agreement 

Mean percentages of agreement were computed for each of four 
raters for each of 10 tapes. These means were then consid- 
ered both independently and in the form of sub and total 
scores within instrument domains. When computing the arith- 
metic mean for percentages (or any other ratio), a value of 
.50 indicates the numerator was twice as small as the denom- 
inator, a value of .25 four times as small and a value of 
.10, ten times as small. If the percentages are simply av- 
eraged, the mean would be .283, and would suggest agreement 
about une fourth the time. This is incoriect, since the ac- 
tual multiples are respectively 2, 4 and .0 (mean = 5.33). 
Thus, the mean percentage should be .183 or agreement about 
one fifth, not one fourth of the time. To avoid this prob- 
lem, the log to the base e of each individual percentage was 
used in the computation of mean logs. The anti-log of this 
mean was then taken to define the percentage of agreement 
for the "average" rater. 

- 18 - 

ERIC 22 



Traditionally, agreement percentages are computed at the 
item level, assigning an agreement percentage of 100 for 
each item of agreement and of 0.00 for each disagreement and 
computing the mean. Although appropriate for yes/no and 
RONSI items, this technique must underestimate interrater 
agreement for the S and 0 elements of the ESON scale. Obvi- 
ously, E(xtensively) agrees with S( ignif icantly) to a great- 
er degree than it does with N(egligible) . Therefore, for 
all ESON scaled items, adjacent pairs (E and S, S and 0, 0 
and N) were considered to agree 50%. All other disagree- 
ments were assigned a value of 0% agreement. Such a scaling 
does not specify a model of instruction, but rather credits 
various possible combinations 

Since both item level (for descriptive purposes) and 
score level (for purposes of ranking CBT packages) are of 
interest here, agreement was computed at both levels. At 
the item level, percentage of agreement represents the mean 
agreement among four raters (six rater pairs). For compos- 
ite scores, agreement represents the mean percentage across 
all items (i), all rater pairs (k r ) and all subjects (j) for 
each score, with item level agreement defined as: 



xi - 



x (i+l) 



1.0 



R = 



Range 
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SCORE RELIABILITY 

For the CCCEM, the average reliability of an individual 
judge is of interest, since each CBT package will be rated 
by only one judge. Thus, mean differences in the ratings of 
different judges is of great importance* Also, one wishes 
the results of these analyses to be generalizable to other 
samples of raters using the same scale with a similar sample 
of targets. Therefore, between rater variance is here in- 
cluded in the error term, which gives the expected reliabil- 
ity of an "average" judge and reduces the reliability esti- 
mate toward its lower bound (Shrout & Fleiss, 1978) • The 
ICC coefficient is computed using an Analysis of variance 
table: with between target variance (BMS) treated as the 
"True variability" and within cell variance (WMS - combining 
instrument, time and rater error) treated as error: 

BMS - WMS 

2,0 R = 

BMS + (k-l)WMS 

where: 

BMS = between targets mean square 
WMc = error or within targets mean square 
k = number of raters/judges 

A second, equally conservative reliability estimate was 
produced by taking the median Pearson Product Moment Corre- 
lation Coefficient among each pair of raters scores for all 
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10 CBT packages. This estimate contains different sources 
of error than the preceding, and provides a good indication 
cf the consistency with which different raters rank the same 
CBT packages in the same order from high to low. 

SAMPLE 

Educational software at present provides only limited exam- 
ples in the academic disciplines. Much software is dedicat- 
ed to various word processing and database management sys- 
tems in addition to some traditional areas of skill 
development such as typing. Therefore, a sample was chosen 
to represent certain strata (characteristics) present in the 
population of all CBT package courseware. Table 3 shows the 
selected sample An attempt was made to include several com- 
monly occuring subject areas, several major producers of 
software, several contact time intervals and several levels 
of instruction design complexity. At the time of sampling, 
neither the range nor variability of CCCEM scores was known. 
Pour experienced raters evaluated each CBT package. 
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TABLE 3 

Sample of Software Chosen as Targets fJor Reliability Study 



Publisher Title/Subject Contact Efficacy 

Time Score 



Blue Chip 


American Dream/Management 


8 


55 


ATI 


DBASE III 


2 


44 


ATI 


SUPERCALC 


2 


35 


Compre 


Reading and Understanding 


20 


33 


CDEX 


IBM PC DOS 


4 


47 


CDEX 


LOTUS III 


2 


49 


David 


Sharpening Your Executive 
Writing Skills 


3 


54 


Thoughtware 


Managing Stress 


4 


46 


Thoughtware 


Improving Employee Performance 


4 


50 


QED 


Typing Made Easy 


4 


58 



RESULTS 



ITEM LEVEL AGREEMENT 

ThG following discussion considers only the 192 items 
upon which disagreement could occur. The remaining items 
(24 scalable) derive from CBT package documentation and are 
not subject to rater disagreement. 

Within these data, one major source of error (disagree- 
ment) exists that usually is not present during evaluations. 
To obtain an adequate number of experienced evaluators for 
the reliability study, it proved necessary to include evalu- 
ators having different areas of expertise (2 MBAs, 1 Psy- 
chologist, 1 Linguist). Thus, for every CBT package evalu- 
ated, at least one evaluator was out of field. This is 
particularly important when one considers that agreement 
among three of four evaluators produces a percentage of 50% 
(not 75%). This results because six possible agreement 
pairs occur for four evaluators (1 & 2, 1 & 3, 1 & 4, 2 & 3, 
2 & 4 and 3 & 4). Thus, if one evaluator codes R(onsi) 
while three code O(ptional), only three of the six pairs 
agree (50%) although 75% of the evaluators agree that 
O(ptional) is correct. Assigning partial agreement (50%) 
for adjacent ESON scale points reduces this problem some- 
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what f but for all agreements, the percentage obtained is f 
both conceptually and realistically, something of an under- 
estimate. Table 4 shows how many raters must agree to 
produce various percentages of agreement. It is clear from 
Table 4 that 75% agreement is quite good. Thus f in the fol- 
lowing discussions, 75% or greater agreement will be consid- 
ered adequate. The assignment of partial agreements for 
ESON scalings increases the percentage of agreement somewhat 
(for four raters, the increase averages approximately 10% to 
15%, since many disagreements are not adjacent scale 
points) . 

TABLE 4 

Interpreting Agreement Percentages 



Number of Evaluators Percentage 
Reaching Agreement of Agreement 



3 of 4 50% 

4 of 5 60% 

5 of 6 67% 

6 of 7 71% 

7 of 8 75% 



Another, unanticipated source of error resulted from a 
definition clarification session which took place during the 
conduct of reliability evaluations by CBT package evalua- 
tors. Definitions were clarified on several heavily weight- 
ed items. Later Discussions with specific evaluators indi- 
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cated that, following the session, their codes would more 
closely corresponded with other evaluators for certain items 
where differences exist in the reliability data reported 
here* It was deemed inappropriate, however, to make any ad- 
justments in the data for these situations . Therefore, the 
reliabilities reported are conservative for yet another rea- 
son. 

Table 5 shows the item level agreement percentages zor 
the 192 scalable items upon which disagreement may exist. 
In general, agreements are high, with only 20,8% showing 
agreement among fewer than seven of eight evaluators. Some 
areas of relatively great disagreement may be attributed to 
evaluators working in areas outside their expertise. In 
particular, whether books are an Extrinsic (67%) or a Sup- 
plemental (55%) aspect of the resource scope; the frequency 
of Context Sensitive Questions (72%); and the techniques 
used for Response Judging (62%): Only One Response (57%), 
Wait for Correct Response (63%), Wrong and Continue (69%), 
Wrong and Hint (72%), Give Correct and Go (56%) • 

Some items appear to require, and since have undergone 
clarification or simplification: (1) Documentation - In- 
structor (50%), Student (50%) and whether it is in the form 
of Manuals, Tutorials or Both (45%), (2) Varying - Text 
Fonts (73%) and Formats (52%), (3) Questioning - Continua- 
tion Response (69%) and On-Line Help (65%), (4) Graphics - 
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whether Color (58%), Windowing (66%) or Illustration (66%), 
and, (5) Sequencing, where several items fall below 75% 
agreement . 

At least one item appeared difficult for evaluators to 
determine: Security - Student Access (70%). Perceptions 
regarding the preponderence cf certain items also appear to 
differ somewhat, suggesting the need for clearer defini- 
tions: Page Turning (72%) and Questioning (69%) as instruc- 
tion techniques; Forced Waits (70%) and ability of students 
to Quit (60%) lessons at any time; and Feedback (50%): Po- 
sitive (62%), Negative (75%), Neutral (58%). In addition, 
one item showed a relatively low percentage of agreement for 
no apparent reason: preponderance of Computer Tones (53%). 
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TABLE 5 

Item Level Agreement for Four Raters Across Ten CBTs 





CBTs 


Ratrs 


Mean 










Pet. 


Item 


n 


n 


Agree 


COLOR 


10 


4 


1.00 


80 CHARACTERS 


10 


4 


0.90 


40 CHARACTERS 


10 


4 


0.93 


UPPER AND LOWER CASE 


10 


4 


1.00 


PERIPHERALS 


10 


4 


0.93 


PERIPHERALS PRINTER 


10 


4 


0.90 


DOCUMENTATION MANAGER 


10 


4 


0.50 


DOCUMENTATION MANUALS 


10 


4 


0.83 


DOCUMENTATION TABLES OF CONTENTS 


10 


4 


0.80 


DOCUMENTATION INDEX 


10 


4 


1.00 


DOCUMENTATION INSTRUCTOR 


10 


4 


0.50 


DOCUMENTATION MANUALS 


10 


4 


0.88 


DOCUMENTATION TABLES OF CONTENTS 


10 


4 


1.00 


DOCUMENTATION INDEX 


10 


4 


1.00 


DOCUMENTATION STUDENT 


10 


4 


0.50 


DOCUMENTATION MANUALS 


10 


4 


0.45 


DOCUMENTATION TABLES OF CONTENTS 


10 


4 


0.93 


DOCUMENTATION INDEX 


10 


4 


1.00 


MS RECORD KEEPING? 


10 


4 


0.80 


INDIVIDUALS 


10 


4 


0.84 


GROUPS 


10 


4 


0.92 


SUBGROUPS 


10 


4 


1.00 


TESTING HISTORY 


10 


4 


0.88 


UNANTICIPATED RESPONSES 


10 


4 


0.90 


COURSEWARE VALIDITY 


10 


4 


1.00 


MS DATA ENTRY AUTOMATIC 


10 


4 


0.80 


MS DATA ENTRY MANUAL 


10 


4 


1.00 


MS DATA ANALYSIS 


10 


4 


0 . 95 


MS DATA ANALYSIS INTERNAL 


10 


4 


0.95 


MS DATA ANALYSIS TEST 


10 


4 


0.95 


MS DATA ANALYSIS ITEM 


10 


4 


0.95 


MS DATA ANALYSIS EXTERNAL 


10 


4 


0.95 


MS REPORTS 


10 


4 


0.83 


MS REPORTS PRE FORMATTED 


10 


4 


0.83 


MS REPORTS USER FORMATTED 


10 


4 


1.00 


MS MEDIA PAPER BASED 


10 


4 


0.92 


MS MEDIA FLOPPY 


10 


4 


0.72 


MS MEDIA NETWORK 


10 


4 


1.00 


MS MEDIA MAINFRAME 


10 


4 


1.00 


MS ACCCEMSS 


10 


4 


0.75 


MS ACCCEMSS TERMINAL 


10 


4 


0.95 


MS ACCCEMSS MICRO 


10 


4 


0.75 


MS ACCCEMSS HARDCOPY 


10 


4 


0.76 
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Table 5 Continued 
Item Level Agreement for Pour Raters Across Ten CBTs 



CBTs Ratrs Mean 
Pct . 



Item 






n 


n 


Agree 


MS SECURITY 






10 


4 


0.90 


MS SECURITY STUDENT ACCESS 




10 


4 


0.70 


MS ACCCEMSS MANAGER ONLY 






10 


4 


0.75 


MS LEARNING PRESCRIPTIONS 






10 


4 


0.93 


MS LEARNING PRESCRIPTIONS 


FORMATIVE 


10 


4 


0.95 


MS LEARNING PRESCRIPTIONS 


COURSE 


LEVEL 


10 


4 


0.95 


MS LEARNING PRESCRIPTIONS 


MODULE 


LEVEL 


10 


4 


1.00 


MS LEARNING PRESCRIPTIONS 


ADAPTATION 


10 


4 


0.95 


OUTCOME DEFS COGNITIVE 






10 


4 


1.00 


OUTCOME DEFS MEASURED 






10 


4 


0.93 


OUTCOME DEFS AFFECT X T *3 






10 


4 


1.00 


OUTCOME DEFS MEASURED 






10 


4 


1.00 


OUTCOME DEFS PSYCHOMOTOR 






10 


4 


1.00 


OUTCOME DEFS MEASURED 






10 


4 


1.00 


PROCESS DEFS COGNITIVE 






10 


4 


1.00 


PROCESS DEFS DOCUMENTED 






10 


4 


1.00 


PROCESS DEFS AFFECTIVE 






10 


4 


1.00 


PROCESS DEFS DOCUMENTED 






10 


4 


1.00 


PROCESS DEFS PSYCHOMOTOR 






10 


4 


1.00 


PROCESS DEFS DOCUMENTED 






10 


4 


1.00 


RESOURCE SCOPE INTRINSIC 






10 


4 


0.78 


TEXT 






10 


4 


0.75 


GRAPHICS 






10 


4 


0.84 


ANIMATION 






10 


4 


0.88 


SOUND 






10 


4 


0.77 


SOUND ON/OFF 






10 


4 


0.83 


RESOURCE SCOPE EXTRINSIC 






10 


4 


0.77 


BOOKS 






10 


4 


0.67 


LABS 






10 


4 


1.00 


SEMINARS 






10 


4 


1.00 


LECTURES 






10 


4 


1.00 


FIELD TRIPS 






10 


4 


1.00 


HELP SESSIONS 






10 


4 


1.00 


TECHNOLOGICAL 






10 


4 


0.95 


SUPPLEMENTAL 






10 


4 


0.77 


BOOKS 






10 


4 


0.55 


LABS 






10 


4 


1.00 


SEMINARS 






10 


4 


1.00 


LECTURES 






10 


4 


1.00 


FIELD TRIPS 






10 


4 


1.00 


HELP SESSIONS 






10 


4 


1.00 


TECHNOLOGICAL 






10 


4 


0.76 
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Table 5 Continued 
Item Level Agreement for Four Raters Across Ten CBTs 





CBTs 




flea H 








Ppf- 


Item 




n 


An roo 
*Ay L CC 


PAGE TURNING 


10 


4 


0 7? 


TUTORIAL 


10 


4 


0 81 


DRILL AND PRACTICE 


10 


4 


0 QO 


QUESTIONING 


10 


4 


V « U -7 


CONCURRENT TRAINING 


10 


4 


0 Q 1 


PROBLEM SOLVING 


10 


4 


0 Q? 


SIMULATION 


10 


4 
** 


n 77 


MODELING 


10 


4 
** 


n A7 

u « o / 


GAMING 


10 


4 
** 


0 QO 


COOPERATIVE GAMING 


10 


4 
** 


i on 


INQUIRY 


10 


4 




TEXT COLORS 


10 


4 


0 7fi 


TEXT SIZING 


10 


4 


0 ft 1 


TEXT FONTS 


10 


4 


n 7 1 


TEXT FORMATS 


10 


4 
** 


0 5? 


TEXT FLASH OR BLINK 


10 


4 


0 87 


TEXT PRINTOUTS 


10 


4 
** 


0 81 


TEXT SCREEN DUMP 


10 


4 


0 81 


GRAPHICS COLOR 


10 


4 


0 58 


GRAPHICS ANIMATION 


10 


4 


0 84 


GRAPHICS VIDEO 


10 


4 


1.00 


GRAPHICS WINDOWING 


10 


4 


0.66 


GRAPHS, CHARTS ETC 


10 


4 


0.86 


ILLUSTRATIONS 


10 


4 


0.66 


GRAPHICS FLASH OR BLINK 


10 


4 


0.94 


GRAPHICS PRINTOUT 


10 


4 


0.95 


GRAPHICS SCREEN DUMP 


10 


4 


0.95 


SOUNDS COMPUTER TONES 


10 


4 


0.53 


SOUNDS EXTERNAL SOURCES 


10 


4 


0.92 


SOUNDS MUSIC 


10 


4 


0.92 


SOUNDS SYNTHESIZED VOICE 


10 


4 


1.00 


TIMING FORCED WAIT 


10 


4 


0.70 


OVERRIDE 


10 


4 


0.95 


FORCED MOVEMENT 


10 


4 


0.86 


OVERRIDE 


10 


4 


0.97 


DISPLAY INTERRUPT 


10 


4 


0.95 
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Table 5 Continued 



Item Level Agreement for Four 


Raters Across Ten 


CBTs 






CBTs 




Moan 










Item 


xx 




«y tec 


INTERACTION PROMPTS 


10 


4 


0 .77 


INTERACTION FEEDBACK 


10 


4 




INTERACTION FEEDBACK POSITIVE 


10 


4 


0 6 9 

VJ • U Cm 


INTERACTION FEEDBACK NEGATIVE 


10 


4 


0 75 


INTERACTION FEEDBACK NEUTRAL 


10 


4 


0 5ft 

U « JO 


INPUT QUITTING 


10 


4 


o fin 


INPUT RESTART 


10 


4 


0 7ft 


QUESTIONS? 


10 


4 


0 7ft 


QUESTIONS MULTIPLE CHOICE 


10 


4 


0 7ft 


QUESTIONS MATCHING 


10 


4 


0 87 


QUESTIONS TRUE FALSE 


10 


4 


n ftn 


QUESTIONS FILL BLANK 


10 


4 


0 75 


QUESTIONS NUMERIC RESPONSE 


10 


4 


0 82 


QUESTIONS GRAPHIC 


10 


4 


1 00 


QUESTIONS FREEFORM TEXT 


10 


4 


0 84 


QUESTIONS CONTINUATION RESPONSE 


10 


4 


0 69 


QUESTIONS ON LINE HELP 


10 


4 


0 65 


QUESTIONS CONTEXT SENSITIVE 


10 


4 


0.72 


RESPONSE JUDGING 


10 


4 


0 62 


ONLY ONE RESPONSE 


10 


4 


0 . 57 


SLIGHT ERRORS 


10 


4 


0 . 81 


SENTENCE PARSING 


10 


4 


1 . 00 


LINGUISTIC 


10 


4 


0.92 


MATHEMATICAL 


10 


4 


0 . 95 


NATURAL LANGUAGE 


10 


4 


1 . 00 


WAIT FOR CORRECT RESPONSE 


10 


4 


0 . 63 


WRONG AND CONTINUE 


10 


4 


0 . 64 


;trong AND HINT 


10 


4 


0.72 


WRONG AND HELP 


10 


4 


0.96 


WRONG AND TUTORIAL 


10 


4 


0.97 


WRONG AND OFF LINE 


10 


4 


1.00 


SCORE AND GO 


10 


4 


0.84 


GIVE CORRECT AND GO 


10 


4 


0.56 


STUDENT TYPES CORRECT RESPONSE 


10 


4 


0.72 


ANSWER KEY PROVIDED 


10 


4 


0.82 




Table 5 Continued 
Item Level Agreement for Pour Raters Across Ten CBTs 







CBTS 


Ratrs 


Mean 










Pet. 


Item 




n 


n 


Agree 


SEUQENCE 


LINEAR 


10 


4 


0.62 


SEUQENCE 


DUMMY KEY 


10 


4 


0.57 


SEUQENCE 


BRANCHING 


10 


4 


0.56 


SEUQENCE 


DIRECTION 


10 


4 


0.53 


SEUQENCE 


BACKWARD? 


10 


4 


0.60 


SEUQENCE 


ONE PAGE BACKWARD 


10 


4 


0.63 


SEUQENCE 


MULTIPLE PAGES BACKWARD 


10 


4 


0.92 


SEUQENCE 


FOWARD 


10 


4 


0.73 


SEUQENCE 


ONE PAGE FORWARD 


10 


4 


0.80 


SEUQENCE 


MULTIPLE PAGES FOWARD 


10 


4 


0.95 


SEUQENCE 


MASTERY 


10 


4 


0.92 


SEUQENCE 


CREDIT FOR MASTERY 


10 


4 


0.94 


SEUQENCE 


ADAPTIVE 


10 


4 


0.88 


SEUQENCE 


BROWSE 


10 


4 


0.70 


SUPPLEMENTAL 


10 


4 


0.77 


BOOKS 




10 


4 


0.55 


LABS 




10 


4 


1.00 


SEMINARS 




10 


4 


1.00 


LECTURES 




10 


4 


1.00 


FIELD TRIPS 


10 


4 


1.00 


HELP SESSIONS 


10 


4 


1.00 


TECHNOLOGICAL 


10 


4 


0.76 


DESIGN FEATURES SCREEN LAYOUT 


10 


4 


0.95 


DESIGN FEATURES LESSON FLOW 


10 


4 


1.00 


DESIGN FEATURES COLOR 


10 


4 


0.80 


DESIGN FEATURES SOUND 


10 


4 


0.75 


TESTING RANDOM 


10 


4 


0.84 


TESTING FIXED 


10 


4 


0.76 


TESTING NONTRAD I T IONAL 


10 


4 


0.91 


TEXT EDITING CHARACTER 


10 


4 


0.53 


TEXT EDITING LINE 


10 


4 


0.97 


TEXT EDITING SCREEN 


10 


4 


0.95 



Scale Level Percentages of Agreement 

Tables 6 and 7 show that at both the scale and subscale 
levels, mean agreement percentages are quite high, with only 
four of the Instruction subscales (Table 7): Planning (66%), 
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Student Documentation (66%) , Interaction (70%) and Sequenc- 
ing (70%) falling noticably below the 75% criterion. For 
total Scales, Physical (91%) , Presentation (79%), Management 
(87%) and Instruction (80%) agreement percentages all fall 
above the. criterion and indicate agreement among approxi- 
mately nine of ten raters. 
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TABLE 6 

Agreement Percentages Across Scores and Subscores 



Mean Percent of 
CITAR CCCEM Scores Agreement 

(4 raters, 10 CBTs) 



TOTAL EFFICACY SCORE 0.83 

Physical Score 0.91 

Physical Screens NA* 

Physical Peripherals 0*91 

Physical Concurrent Applications NA 

Presentation Score 0.79 

Presentation Interface NA 

Presentation Text 0,73 

Presentation Graphics 0,81 

Presentation Sound 0,79 

Presentaiton Design Features 0.85 

Presentation Text Editing 0.77 

Management Score 0.87 

Management Documentation 0.77 

Management Record Keeping 0.87 

Management Data Entry/Analysis 0.92 

Management Reports 0.87 

Management Media 0.89 

Management Access NA 

Management Security 0.78 

Management prescriptions 0.95 

Management Browse/Sequencing 0.96 

Instruction Score 0.80 

Planning Subscore 0.66 

Objectives Subscore 0.99 

Techniques Subscore 0.80 

Interaction Subscore 0.73 

Resource Scope Subscore 0.84 



* The items designated NA are determined from 

documentation by project manager prior to evaluation. 
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TABLE 7 

Agreement Percentages Across Instruction Subscores 



Mean Percent of 



CITAR CCCEM Score Agreement 

(4 raters, 10 CBTs) 



Instruction Score 0.80 

Planning Subscore 0.66 

Instructive Student Documencation 0.66 

Instructive Task Analysis , NA 

Objectives Subscore 0.99 

Instructive Stated Goals NA 

Instructive Defined Outcomes 0.98 

Instructive Defined Processes 1.00 

Techniques Subscore 0.80 

Instructive Techniques 0.83 

Instructive Presentation 0.74 

Instructive Timing 0.87 

Interaction Subscore 0.73 

Instructive Judging Wrong Response 0.74 

Instructive Testing 0.82 

Instructive Interaction 0.70 

Instructive Questioning 0.75 

Instructive Sequencing 0.70 

Resource Scope Subscore 0.84 

Instructive Intrinsic Resource Scope 0.79 

Instructive Extrinsic Resource Scope 0.90 

Instructive Supplemental Materials 0.85 



* The items designated NA are determined from 

documentation by project manager prior to evaluation. 
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RELIABILITY OF MAJOR SCORES 

Regarding the ability of CCCEM scores to consistently dis- 
criminate across raters, Table 8 shows that the Efficacy 
score (.65, .11), the Management score (.67, .71) and the 
Instruction score (.66, .73) exhibit reasonable levels of 
reliability. Given the extremely conservative nature of 
these reliability estimates, there should be no reason to 
avoid the use of these scales for purposes of grouping CBT 
packages into four categories: Outstanding, Excellent, Mod- 
erate and Questionable. In order for * measure to differen- 
tiate consistently among CBT packages, i is necessary for 
those CBT packages to differ on the items of interest. 
Where useful differences are lacking, the instrument may ex- 
hibit high interrate agreement but cannot exhibit high reli- 
ability. The ranges of mean scores (mean of 4 raters) in 
Table 8 show that limited variability contributed to the 
lower reliability estimates for the Physical score (.55, 
.53) and the extremely low estimates obtained for the Pres- 
entation score (.17, .11). This also appears the major 
source of unreliability for the Planning subscore of in- 
struction (.29, .47), although rater disagreement also con- 
tributed here (.66) and in the Interaction subscore (.73). 
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TABLE 8 

Reliability of Major and Subscales 



Range of Reliability 
CITAR Mean Pet. 



Score/Subscore 


Scores 


ICC 


Tst^tst 


Agrmt 


Efficacy 


34 - 


57 


.65 


.77 


.83 


Physical Characteristics 


66 - 


80 


.55 


.53 


.51 


Presentation Aspects 


38 - 


56 


.17 


.11 


.79 


Management Structure 


2 - 


49 


.67 


.71 


.87- 


Instruction Aspects 


35 - 


64 


.66 


.73 


.80 


Planning Subscore 


18 - 


30 


.29 


.47 


.66 


Techniques Subscore 


35 - 


81 


.45 


.54 


,9° 


Objectives Subscore 


13 - 


83 


.93 


.94 


.8>> 


Interaction Subscore 


14 - 


59 


.63 


.63 


.73 


Resource Scope Subscore 


48 - 


95 


.41 


.53 


.84 



Another interesting aspect of the scores in Table 8, is 
the distance from optimum obtained by the highest scoring 
CBT packages for most scales. For four of tho five major 
scores, no CBT package achieved two-thirds of the optimum 
score (.67). Only in the Physical score did any reach four- 
fifths of optimal (.80). Also, under Management, although 
considerable variability occurred, no CBT package even 
reached 50% of optimality. Among Instruction subscores, 
Planning and Interaction are particularly lacking among the 
CBT packages. It should be noted here that several subs- 
cores or items (e.g. task analysis) were found perhaps only 
once among the ten CBT packages included in this study. 
This despite the fact that the sample was chosen to include 
as many characteristics of the CCCEM system as possible. 




ASSIGNMENT OP VALUES TO SCORES 

Values (Good/Bad, High/Low) may be assigned to scores at- 
tained by CBT packages on the various subscales based upon 
either of two criteria: (1) Relative Performance or {2) Ab- 
solute Performance. Since each score represents a propor- 
tion of the optimal, either technique is applicable. For 
example, given an optimum score of 100, one might consider 
any CBT package attaining over 75% be defined as Outstand- 
ing. Or, given the relative scores of all CBT packages cur- 
rently evaluated (n=110), one might consider the top 10/15 
percent to be outstanding. 

So few CBT packages currently approach optimum, it ap- 
pears most appropriate to develop categories using relative 
percentiles : 

1. OUTSTANDING - 85th to 99th percentile, 

2. EXCELLENT - 60th to 84th percentile, 

3. MODERATE - 40th to 59th percentile, and 

4. QUESTIONABLE- 1st to 39th percentile. 

Hopefully, over time, current CBT packages will be super- 
ceded by more sophisticated and efficacious software. Based 
upon the first 180 CBT packages, the following outpoints ap- 
pear appropriate (Appendix A contains source data for cut- 
point development). 
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TABLE 9 
Cut Scores for Rating CTBs 



Below 



Score 


Average 


Average 


Excellent 


Outstanding 


Efficacy 


0-40 


41 - 46 


47 - 50 


51 + 


Instruction 


0-47 


48 - 54 


55 - 63 


64 + 


Management 




0-29 


30 - 50 


51 + 


Presentation 


Inadequate Reliability for Ratings 


Physical 


0-65 


66 - 71 


72 - 77 


78 + 



COMPARISON OF RATER PERCEPTIONS AND CCCEM SCORES 
In order to determine whether a gain in reliability (consis- 
tency) occurs by using relatively objective criteria such as 
those in the CCCEM model over pure evaluations by raters, 
the four evaluators also produced a quality rating for each 
CBT package corresponding to the four major CCCEM scores: 
Instruction, Management, Presentation and Physical. Ratings 
were based on a scale from zero to 10, where zero represents 
non-existent or useless and 10 represents optimal. Score 
based reliabilities using formula 1.0 were then computed for 
each of these and compared with similar results produced by 
the "CCCEM scores. ICC estimates using formula 2.0 were also 
produced and compared between the two sources of informa- 
tion. As Table 10 shows, for agreement percentages, those 
produced by the CCCEM, even at the item level, average about 
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20 percentage points higher than those produced by pure ra- 
ter perceptions. Perceptual disagreements among raters 
(shown in the ICC columns of Table 10) , tend to invalidate 
an instrument. None of the perceptual scores showed the 
ability to consistently discriminate among CBT packages. 
With reliability estimates ranging from .08 to .23, it is 
not possible to determine which CBT package a particular ra- 
ter is evaluating, since the within package variabilty is 
almost as the between package variability. Based upon this 
evidence, one may conclude that it is not possible to com- 
pare evaluations of two different software packages by two 
different evaluators based upon purely perceptual or opin- 
ion. The CCCEM, on the other hand, allows one to compare 
evaluations made by one rater with those of another on dif- 
ferent software. Interestingly, Presentation was the least 
reliable score both perceputally and objectively. 



TABLE 10 

Consistency Estimates from Perceptions and CCCEM Scores 





Agreement 


ICC 




Score 


Perceptions 


CCCEM 


Perceptions 


CCCEM 


Efficacy 


.69 


-83 


.18 


.65 


Instruction 


.61 


.80 


.23 


.66 


Management 


.64 


.87 


.10 


.67 


Presentation 


.65 


.79 


.08 


.17 


Physcial 


.69 


.91 


.11 


.55 
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Appendix A 

DISTIBUTION OP CITAR SCORES ATTAINED BY CURRENT 

CBT PACKAGES 

The following tables show the scores obtained by all CBT 
packages evaluated by CITAR as of August, 1987 on a scale 
from zero to 100. Table 10 indicates that although Efficacy 
scores exhibit variability (s x = 8.416), the mean (43.36) 
and range (20 - 64) of scores shows considerable room for 
improvement. Physical scores (Table 11) exhibit limited 
variability (s x = 6.42) sr-il show relatively higher propor- 
tions of optimality than other scales (mean = 71.11). Pres- 
entation scores (Table 12) range substantially (20 to 65). 
Management scores (Table 13) suggest that few CBT packages 
exhibit many management system characteristics, with 84% re- 
ceiving less than 10% of the possible score. Of those hav- 
ing management systems, fewer than 10% exhibit as much as 
40% of the optimum capicity. Table 14 shows the overall in- 
struction score, which ranges from 17 to 77, has a mean of 
51.18 and exhibits adequate variability (s x = 11.99). 
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TABLE 11 

Total Efficacy Score - Frequencies for 180 CBTs 



Cumulative 
Score Frequency Percent Percent 



9 n 
z u 


i 
j_ 


9 ^ 
Z J 


i 


Z 3 




97 
Z / 


i 

L 


ZO 


«j 


9Q 

Z 27 


1 


3n 


1 

-L 




D 


39 
o z 


-3 


33 






Q 


35 


9 
z 


3fi 


q 


37 


Q 


3ft 


4 


3Q 


o 


40 


a 

u 


41 


1 1 

X J. 


42 


A 


43 


c; 
j 


44 


Q 
O 


45 


7 


4fi 




47 


14 


48 


10 


49 




5tt 


7 


51 


u 




9 
z 


53 


4 


54 


5 


55 


3 


56 


2 


57 


3 


. 59 


2 


60 


1 


62 


1 


63 


1 


64 


2 


TOTAL 


180 


MEAN 


43.356 


SKEWNESS 


-.038 


STD DEV 


8.416 



. o 


. o 


a 
. u 


± • ± 


. u 


1 7 
x • / 
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« u 


9 9 
Z . Z 


1 7 
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3 Q 
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3 3 
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1 7 ft 
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97 ft 
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3fi 7 

JU ♦ / 
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49 ft 
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47 8 


4 . 4 


52 ? 


3 . 9 


56 1 
J U » X 


2 . 8 

£t • SJ 


58 Q 


7.8 


66.7 


5.6 


72.2 


O . X 


7fl ^ 
/ O • J 


3.9 


82.2 


3.3 


35.6 


1.1 


86.7 


2.2 


88.9 


2.8 


91.7 


1.7 


93.3 


1.1 


94.4 


1.7 


96.1 


1.1 


97.2 


.6 


97.8 


.6 


98.3 


.6 


98.9 


1.1 


100.0 


100.0 





MEDIAN 44.000 
RANGE 44.000 
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TABLE 12 

Total Physical Score - Frequencies for 180 CBTs 



Cumulative 
Score Frequency Percent Percent 



52 


1 


57 


8 


63 


2 


64 


5 


66 


49 


69 


5 


70 


5 


72 


53 


73 


2 


75 


2 


76 


6 


78 


30 


82 


8 


88 


4 


TOTAL 


180 



MEAN 71.106 
SKEWNESS .031 
STD DEV 6.424 



.6 


.6 


4.4 


5.0 


1.1 


6.1 


2.8 


8.9 


27.2 


36.1 


2.8 


38.9 


2.8 


41.7 


29.4 


71.1 


1.1 


72.2 


1.1 


73.3 


3.3 


76.7 


16.7 


93.3 


4.4 


97.8 


2.2 


100.0 


100.0 





MEDIAN 72.000 
RANGE 36.000 
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TABLE 13 

Total Presentation Score - Frequencies for 180 CBTs 



Cumulative 



Score 


Fr eouencv 




Dp rnpn 
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• u 


a 
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• u 
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27 
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2 
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32 


4 


2 . 2 




33 


3 
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3 


1.7 
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35 


5 
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36 


5 
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37 


9 
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38 
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4 . 4 
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40 


3 


1.7 
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41 


9 
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30 6 


42 


7 
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43 


1 
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42 . 8 
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45 


11 
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52 . 8 


46 


8 


4.4 


57 . 2 


47 


3 


1.7 


58 . 9 


48 


5 
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61. 7 


49 


7 


3.9 


65.6 


50 


8 


4.4 


70.0 


51 


6 


3 . 3 


73 3 


52 


14 


7.8 


81.1 


53 


3 


1.7 


82.8 


54 


8 


4.4 


87.2 


55 


11 


6.1 


93.3 


56 


2 


1.1 


94.4 


57 


2 


1.1 


95.6 


58 


1 


.6 


96.1 


59 


2 


1.1 


97.2 


60 


1 


.6 


97.8 


64 


1 


.6 


98.3 


65 


3 


1.7 


100.0 


TOTAL 


180 


100.0 





MEAN 45.278 MEDIAN 45.000 

SKEWNESS -.175 RANGE 45.000 

STD DEV 8.202 
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TABLE 14 

Total Management Score - Frequencies for 180 CBTs 



Cumulative 
Score Frequency Percent Percent 



1 


51 


2 


73 


3 


15 


4 


10 


6 


1 


7 


1 


21 


1 


26 


1 


28 


1 


34 


1 


36 


1 


37 


2 


38 


3 


39 


3 


41 


1 


43 


2 


44 


4 


45 


2 


46 


2 


47 


1 


48 


2 


50 


1 


54 


1 


TOTAL 


180 



MEAN 8.206 
SKEWNESS 1.979 
STD DEV 14.614 
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28.3 


40.6 


68.9 


8.3 


77.2 


5.6 


82.8 


.6 


83.3 


.6 


83.9 


.6 


84.4 


.6 


85.0 


.6 


85.6 


.6 


86.1 


.6 


86.7 


1.1 


87.8 


1.7 


89.4 


1.7 
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.6 


91.7 


1.1 
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1.1 


96.1 


1.1 


97.2 


.6 


97.8 


1.1 


98.9 


.6 


99.4 


.6 


100.0 


100.0 





MEDIAN 2.000 
RANGE 53.000 
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TABLE 15 

Total Instruction Score - Frequencies for 180 CBTs 



Cumulative Cumulative 
Score Freq Percent Percent Score Freq Fercent Percent 
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Appendix B 



ITEM CONTENT OF SCALES AND SDPSCALES 

PHYSICAL SUBSCALES 
PHYSICAL SCREENS 



PA01 


HARD DISKS 


PA02 


RESOLUTION GIVEN 


PA03 


BOARDS 


PA04 


TWO SCREENS 


PA05 


COLOR 


PA06 


80 CHARACTERS 


PA07 


40 CHARACTERS 


PA08 


UPPER AND LOWER CASE 



PHYSICAL PERIPHERALS 



PB01 


PERIPHERALS 




PB02 


PERIPHERALS 


PRINTER 


PB02 


PERIPHERALS 


PLOTTER 


PB04 


PERIPHERALS 


EXT HOST 


PB05 


PERIPHERALS 


AUDIOTAPE 


PB06 


PERIPHERALS 


VIDEODISC 


PB07 


PERIPHERALS 


SLIDE 


PB08 


PERIPHERALS 


VOICE RECOGNITION 


PB09 


PERIPHERALS 


CD-ROM 


PB10 


PERIPHERALS 


GAME ADAPTOR 


PB11 


PERIPHERALS 


MODEM 



Presentation Interface 



PJ01 INTERFACE MOUSE 

PJ02 INTERFACE TOUCH 

PJ03 INTERFACE KEYBOARD 

PJ04 INTERFACE LIGHT PEN 

PJ05 INTERFACE BIT/GRAPHICS PAD 

PJ06 INTERFACE MICROPHONE 



PRESENTATION SDBSCORES 



Presentation Text 



PDOi TEXT COLORS 

PD02 TEXT SIZING 

PD03 TEXT FONTS 

PD04 TEXT FORMATS 

PD05 TEXT FLASH OR BLINK 

PD06 TEXT PRINTOUTS 

PD07 TEXT SCREEN DUMP 

Presentation Graphics 

PE01 GRAPHICS COLOR 

PE02 GRAPHICS ANIMATION 

PE03 GRAPHICS VIDEO 

PE04 GRAPHICS WINDOWING 

PE05 GRAPHICS FLASH OR BLINK 

PE06 GRAPHICS PRINTOUT 

PE07 GRAPHICS SCREEN DUMP 

PRESENTATION SOUNDS 

PFC1 SOUNDS COMPUTER TONES 

PF02 SOUNDS EXTERNAL SOURCES 

PF03 SOUNDS MUSIC 

PF04 SOUNDS SYNTHESIZED VOICE 

Presentation Design Features 

PG01 DESIGN FEATURES SCREEN LAYOUT 

PG02 DESIGN FEATURES LESSON FLOW 

PG03 DESIGN FEATURES COLOR 

PG04 DESIGN FEATURES SOUND 

Presentation Text Editing 

PH01 TEXT EDITING CHARACTER 

PH02 TEXT EDITING LINE 

PH03 TEXT EDITING SCREEN 



9 

ERJC 



- 48 - 

52 



MANAGEMENT SOBSCORES 



Management Documentation 



MA01 DOCUMENTATION MANAGER 

MAO 2 DOCUMENTATION MANUALS 

MA03 DOCUMENTATION TABLES OF CONTENTS 

MAO 4 DOCUMENTATION INDEX 

MAO 5 DOCUMENTATION INSTRUCTOR 

MAO 6 DOCUMENTATION MANUALS 

MA07 DOCUMENTATION TABLES OF CONTENTS 

MA08 DOCUMENTATION INDEX 

Management Record Keeping 

MB01 MS RECORD KEEPING? 

MB05 TESTING HISTORY 

MB06 UNANTICIPATED RESPONSES 

MB07 COURSEWARE VALIDITY 

Management Data Analysis 

MC01 MS DATA ENTRY AUTOMATIC 

MC02 MS DATA ENTRY MANUAL 

MC03 MS DATA ANALYSIS 

MC04 MS DATA ANALYSIS INTERNAL 

MC05 MS DATA ANALYSIS TEST 

MC06 MS DATA ANALYSIS ITEM 

MC07 MS DATA ANALYSIS EXTERNAI 

Management Reports 

MD01 MS REPORTS 

MD02 MS REPORTS PRE FORMATTED 

MD03 MS REPORTS USER FORMATTED 

Management Media 

ME01 MS MEDIi; PAPER BASED 

ME02 MS MEDIA FLOPPY 

ME03 MS MEDIA NETWORK 

ME04 MS MEDIA MAINFRAME 

Management System Access 

ME05 MS ACCCEMSS 

ME06 MS ACCCEMSS TERMINAL 

ME07 MS ACCCEMSS MICRO 

ME08 MS ACCCEMSS HARDCOPY 

ME09 MS ACCCEMSS MANAGER ONLY 
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Management System Security 



MF01 MS SECURITY 

MF02 MS SECURITY STUDENT ACCCEMSS? 

Management Learning Prescriptions 

MG01 MS LEARNING PRESCRIPTIONS 

MG02 MS LEARNING PRESCRIPTIONS FORMATIVE 

MG03 MS LEARNING PRESCRIPTIONS COURSE LEV 

MG04 MS LEARNING PRESCRIPTIONS MODULE LEV 

MG05 MS LEARNING PRESCRIPTIONS ADAPTATION 

Management Instructor Browse 

MH01 INSTRUCTOR BROWSE 

KH02 INSTRUCTOR' SEQUENCING 

MHO 3 INSTRUCTOR GROUP 

MH04 INSTRUCTOR INDIVIDUAL 

MHO 5 OPTIONAL SEQUENCE TO HELP 

MHO 6 OPTIONAL SEQUENCE TO HINT 

MH07 OPTIONAL SEQUENCE TO Tt'/.?ORIAL 

MHO 8 OPTIONAL SEQUENCE TO TEST 

MH09 OPTIONAL SEQUENCE TO MENU 

MH10 OPTIONAL SEQUENCE TO OFF LINE 

MH11 TESTING VARIABLE SCORING OPTIONS 

MH12 TESTING UNOBTRUSIVE 

INSTRUCTION SUBSCORES 

Instruction Student Documentation 

IA01 DOCUMENTATION STUDENT 

I AO 2 DOCUMENTATION MANUALS 

IA03 DOCUMENTATION TABLES OF CONTENTS 

IA04 DOCUMENTATION INDEX 

Instruction Defined Goals 

IC01 OBJECTIVES ENTRY PROFICIENCY 

IC02 OBJECTIVES EXIT PROFICIENCY 

Instruction Defined Outcomes 

ID01 OUTCOME DEFS COGNITIVE 

ID02 OUTCOME DEi'S MEASURED 

ID03 OUTCOME DEFS AFFECTIVE 

ID04 OUTCOME DEFS MEASURED 

ID05 OUTCOME DEFS PSYCHOMOTOR 

ID06 OUTCOME DEFS MEASURED 
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Instruction Defined Processes 



IE01 PROCESS DEPS COGNITIVE 

IE02 PROCESS DEPS DOCUMENTED 

IE03 PROCESS DEPS AFFECTIVE 

IE04 PROCESS DEPS DOCUMENTED 

IE05 PROCESS DEPS PSYCHOMOTOR 

IE06 PROCESS DEPS DOCUMENTED 



Instruciton Techniques 



1101 


PAGE TURNING 


1102 


TUTORIAL 


1103 


DRILL AND PRACTICE 


1104 


QUESTIONING 


1105 


CONCURRENT TRAINING 


1106 


PROBLEM SOLVING 


1107 


SIMULATION 


1108 


MODELING 


1109 • 


GAMING 


1110 


COOPERATIVE GAMING 


1111 


INQUIRY 



Instruction Physical Presentation 



IJ01 GRAPHS, CHARTS ETC 

IJ02 ILLUSTRATIONS 

Instruction Timing 

IK01 TIMING FORCED WAIT 

IK02 OVERRIDE 

IK03 FORCED MOVEMENT 

IK04 OVERRIDE 

IK05 DISPLAY INTERRUPT 



Instruction Judging Incorrect Responses 



IN01 


WAIT FOR CORRECT RESPONSE 


IN02 


WRONG AND CONTINUE 


IN03 


WRONG AND HINT 


IN04 


WRONG AND HELP 


IN05 


WRONG AND TUTORIAL 


IN06 


WRONG AND OFF LINE 


IN07 


SCORE AND GO 


IN08 


GIVE CORRECT AND GO 


IN09 


STUDENT TYPES CORRECT RESPONSE 


IN10 


ANSWER KEY PROVIDED 



Instruction Testing 

IP01 TESTING RANDOM 

IP02 TESTING FIXED 

IP03 TESTING NONTRADITIONAL 
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Instruction Interaction 



IL01 INTERACTION PROMPTS 

IL02 INTERACTION FEEDBACK 

IL03" INTERACTION FEEDBACK 

IL04 INTERACTION FEEDBACK 

IL05 INTERACTION FEEDBACK 

IL06 INPUT QUITTING 

IL07 INPUT RESTART 

IL08 RESPONSE JUDGING 

IL09 ONLY ONE RESPONSE 

IL10 SLIGHT ERRORS 

IL11 SENTENCE PARSING 

IL12 LINGUISTIC 

IL13 MATHEMATICAL 

IL14 NATURAL LANGUAGE 



Instruction Questioning 



IM01 


QUESTIONS? 


IM02 


QUESTIONS 


MULTIPLE CHOICE 


IM03 


QUESTIONS 


MATCHING 


IM04 


QUESTIONS 


TRUE FALSE 


IM05 


QUESTIONS 


FILL BLANK 


IM06 


QUESTIONS 


NUMERIC RESPONSE 


IM07 


QUESTIONS 


GRAPHIC 


IM08 


QUESTIONS 


FREEFORM TEXT 


IM09 


QUESTIONS 


CONTINUATION RESPONSE 


IM10 


QUESTIONS 


ON LINE HELP 


IM11 


QUESTIONS 


CONTEXT SENSITIVE 



Instruction Sequencing 



1001 


SEUQENCE 


LINEAR 


1002 


SEUQENCE 


DUMMY KEY 


1003 


SEUQENCE 


BRANCHING 


1004 


SEUQENCE 


DIRECTION 


1005 


SEUQENCE 


BACKWARD? 


1006 


SEUQENCE 


ONE PAGE BACKWARD 


1007 


SEUQENCE 


MULTIPLE PAGES BACKWARD 


1008 


SEUQENCE 


FOWARD 


1009 


SEUQENCE 


ONE PAGE FORWARD 


1010 






1011 


SEUQENCE 


MASTERY 


1012 


SEUQENCE 


CREDIT FOR MASTERY 


1013 


SEUQENCE 


ADAPTIVE 


1014 


SEUQENCE 


UROWSE 



POSITIVE 
NEGATIVE 
NEUTRAL 
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Instruction Resource Scope Intrinsic 





RESOURCE SCOPE INTRINSIC 


It [)Z 


TEXT 




GRAPHICS 


t u»n /i 

Jl-t U4 


A XT T If TV ITl T /*^VV7 

ANIMATION 


±£ Uj 


SOUND 


ins u rue u ion 


Resource Scope Extrinsic 


IG01 


RESOURCE SCOPE EXTRINSIC 


IG02 


BOOKS 


IG03 


LABS 


TPO A 


o.dM±NARq 


IGUS 


LECTURES 


IbUO 


FIELD TRIPS 


±IjU / 


HELP SESSIONS 


J.v?UO 


TECHNOLOGICAL 


insutucuion 


buppiemental 


IH01 


SUPPLEMENTAL 


IH02 


BOOKS 


IH03 


LABS 


IH04 


SEMINARS 


IH05 


LECTiJRES 


IH06 


FIELD TRIPS 


IH07 


HELP SESSIONS 


IH08 


TECHNOLOGICAL 
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